Remote image management system (RIMS)

ABSTRACT

A data processing system is provided which enables an operator to rapidly perform object detection, identification, recognition, and location using remote imagery from Mini Unmanned Air Vehicles when sensor performance is severely limited due to size, weight, and power constraints. The system receives down linked images from an Unmanned Air Vehicle as well as vehicle geographic position and sensor attitude data. The imagery is processed on the ground using detection, identification, recognition and moving target detection algorithms to eliminate clutter and preselect potential objects of interest. The objects of interest are identified by the operator from the preselected list of objects automatically presented to him. The target location is simultaneously calculated for selected objects using the down linked vehicle location and sensor pointing angle and displayed to the operator.

CROSS REFERENCE TO RELATED APPLICATIONS

This Application claims the benefit of U.S. Provisional Application No.60/240,041, filed Oct. 16, 2000.

STATEMENT CONCERNING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable

REFERENCE TO A MICROFICHE APPENDIX

Not Applicable

BACKGROUND OF THE INVENTION

1. Field of Invention

The present invention generally relates to object detection, recognitionand location systems where sensor performance is severely limited due tosize, weight and power constraints.

2. Background of the Invention

Current aerial surveillance and targeting systems are subject to thesame fundamental design constraints: 1) Use of IR detection sensors fornight-time and all weather capability; 2) some form of gimbalingmechanism to reorient the position of the sensor either to seek out newobjects or to maintain the line of sight of the sensor relative to theobject of interest while compensating for the forward motion of theairborne platform or movement of the object itself; 3) stabilization toisolate the sensor from vibration, pitch, yaw and roll motions as wellas air turbulence.

Uncooled IR sensors possess approximately ten percent of the sensitivityto light of a normal day light television camera. Sensitivitylimitations ultimately affect the quality and the detail of the imageswhich may be obtained. Current approaches compensate for this limitationby using multiple sensors arranged in arrays and by cooling the head ofthe IR detector to make it more sensitive to IR radiation. Theseapproaches trade increased sensitivity at the cost of increased size andweight.

The prime example of this is the Forward Looking Infra-Red or FLIR. AFLIR design usually contains other features: a laser range finder, a daytime light television camera, optics to integrate, amplify and clarifyimages from the optical detectors, a means to track and gimbol the FLIRto objects of interest and a means to stabilize the unit from theplatform's engine vibrations, changes in attitude due to aerialmaneuvering and buffeting due to air turbulence. FLIRS are neitherparticularly miniature or light weight. One of the smallest commerciallyavailable FLIRs, the Microstar® manufactured by the FLIR Systems Companyis 34.3 centimeters high and weighs 11.6 kilograms.

Existing airborne detection and targeting systems wheiher manned orunmanned also require the intervention of a trained operator to direct agimbaled sensor to the area of interest, detect and identify the objectsof interest, determine their location, and communicate the informationto a ground based command post for processing and decision making. Thishuman data processing takes place in real time. Support for this missionusually requires a helicopter or fixed wing aircraft and two people—apilot and sensor system operator or an unmanned air vehicle (UAV).

UAVs though unmanned, fall into the size range of manned aircraft. Forexample the General Atomics Predator® has a length of 8.7 meters and awingspan exceeding 15.5 meters. The smaller, General Atomics Prowler hasa length of 4.9 meters and a wingspan of 7.74 meters. These planes arecapable of carrying payloads in the range of 130 to 220 kilogram andremaining aloft for more than 24 hours.

These systems, whether manned or unmanned, require highly trainedpersonnel, are expensive to operate, and at times put the humanoperators in harms way. In addition, they require complex logisticssystems to service, maintain and support the sensor, the airframe whichcarries it, and the air crew. In the case of UAVs, this includes aground control station capable of controlling the UAV, directing thesensor, and receiving transmitted images in real time.

By contrast a MUAV is lighter, 4.5 kilograms or less, has much lowercapital, and operating costs than a conventional aircraft or UAV. Itdoes not require a launch pad or airport, and can be carried in a carfor instant deployment when needed. Like UAVs, it does not put itsoperators in harms way for dangerous missions. MUAV surveillance systemsare however, severely constrained by their own weight, size, and powerlimitations. By way of context, the Microstar® FLIR mentioned above ismore than twice the weight of a MUAV.

In turn, MUAVs tactical capabilities are limited due to the performanceof existing state of the art technology for small, light-weight nightvision, infrared (IR) sensors. To meet weight requirements, Single notarrays of IR sensors must be employed. The sensor must be uncooledbecause use of a cryostatic device to cool the sensor would imposeunacceptable power and weight requirements. Today's state of the artuncooled sensors have limited resolution, requiring a narrow field ofview (FOV) in the range of 15 degrees or a telephoto lens to resolveground objects. In addition, weight and power limitations of an MUAVpreclude the use of a three axis, stabilized, gimbaled platform todirect the sensor.

The use of a fixed, narrow FOV sensor imposes several limitations on thesystem which inhibit its ability to effectively perform its mission. Theoperator must be able to recognize the objects in the FOV of the sensorto discern targets of interest from non targets and differentiate“friend” from “foe”. At the 35 mile per hour baseline speed of currentMUAV's, the operator will experience a “Soda Straw” effect similar tothat experienced in looking out the side of a fast moving vehicle with apair of powerful binoculars. Objects do not remain in the field of viewlong enough for the operator to recognize objects. To illustrate thislimitation, an MUAV equipped with a representative state of the artuncooled IR sensor with a field of view (FOV) of 15 degrees, flying at60 kilometers per hour (17 meters per second) at an altitude of 100meters would have an effective visual area of less than 50 meters indiameter. The time required for a human operator to recognise an objectwithin the FOV is 5 to 9 seconds. Given these parameters, the maximumtime an object would be in the FOV is 3 seconds, less if it is notlocated along the diameter of the FOV. In either case there isinsufficient time in the FOV for operator recognition.

Often, the mission is to search an area for potential objects where noprior information is known about approximate location of objects ofinterest. The vehicle is then forced to fly a search pattern. The timerequired to search even a small area is excessive if the MUAV is forcedto fly at speeds slow enough to enable the operator to recognize targetsbecause of the narrow sensor footprint over the ground. Using the speed,altitude and FOV values of the previous example, and assuming that thetime to execute a 180 degree turn is 30 seconds, the search time for a10 square kilometer area would be almost 5 hours at speeds where imageryis barely recognizable or non-recognizable to the operator.

Once an object of interest is identified, it is desirable to loiter overits location and observe. A circular loiter pattern is not feasible witha fixed camera because the bank angle of the aircraft would be greaterthan 15 degrees. With a FOV of 15 degrees and an altitude of 160 meters,the resulting visual footprint of the state of the art uncooled IRsensor is approximately 80 meters. The object can be observed with aracetrack or figure eight pattern, but these patterns allow the objectto be within the FOV for only approximately 5% of the time in a typicalholding pattern.

Use of an on-board laser range finder is not possible because of payloadweight constraints of MAUVs. Current systems employed on MAUVs thereforedetermine the target location using the Global Positioning Systemcoordinates of the air vehicle and an algorithm which triangulates thelocation of the object of interest using altitude and the LOS angle fromvertical These algorithms assume that the terrain is flat. Significanterrors can be introduced in the situation where the terrain is in factmountainous or hilly.

SUMMARY OF THE INVENTION

The Remote Image Management System (RIMS) described herein eliminatesthe need for manned aircraft, airborne personnel, and their expensivesupport systems. One of its key attributes is that it is small andsimple enough to permit the use of a Mini Unmanned Air Vehicle (MUAV)weighing less than 4.5 kilograms.

The invention includes of both airborne and ground based elements. Theairborne element includes of a device capable of slewing the sensor lineof sight (LOS), a sensor capable of gathering imagery of ground basedobjects, a communication link between the air vehicle and the GroundControl Unit (GCU), and a Flight Management System (FMS) capable ofdetermining the air vehicle location and controlling the airborneprocesses relating to the control of the sensor LOS, the communicationlink, and the air vehicle flight path and attitude.

The ground based element is an operator's GCU including radiotransmitters and receivers comprising a communication link between theGCU and the air vehicle, an image processing unit, comprised of a singleor multiple CPUs and a display and control interface with the operator.In operation, the sensor LOS is directed to an area of interest byeither automatic control of the FMS or by the ground operator. Itsimagery is down linked to the GCU along with data containing the airvehicle location and sensor pointing angle. The GCU data link receivesthe imagery and data and routes it to the image management system. Theimage management system processes the imagery to eliminate the clutterof false targets by performing detection, identification, recognitionand moving target detection algorithms. The purpose for performing thesealgorithms is to identify potential objects of interest by eliminatingnon targets. These objects of interest are highlighted and displayed tothe operator thus reducing the operator workload and enabling him toconcentrate only on potential targets. This reduction of workloadpermits the operator to find targets in real or near real time morereadily than if he were forced to review the entire scene unaided andeliminate the clutter objects himself.

Once an object of interest has been selected, the operator can interfacewith the control panel on the GCU and the target location is noted byperforming calculations in the GCU more fully described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of the Remote Imaging Management System.

FIG. 2 depicts the LOS Slew Device, 10

FIG. 3 is a block diagram of the Image Management Architecture.

FIG. 4 describes the methodology for computing target location usingtriangulation.

DESCRIPTION OF THE PREFERRED EMBODIMENT OF THE INVENTION

Referring to FIG. 1, System Block Diagram, LOS slew device, 10 is alightweight device which enables the Line of Sight of the sensor to bedirected to an area of interest. This device may consist of alightweight mirror rotatable on a single axis or a 3 axis tilt and panmechanism. The control of the LOS slew device may be either automaticthrough preprogrammed direction from the FMS 13 or by control of theoperator on the ground. LOS slew device is described in more detail inFIG. 2. Slewing of the LOS, can be accomplished using the single axismirror technique shown in FIG. 2. A 3 axis pan and tilt mechanism mayalso be employed. A stabilized 3 axis gimbal mechanisms are generallytoo heavy to be accommodated in a MUAV.

The LOS sensor 10, will be coupled with the aircraft attitude in pitch,yaw, and roll (which is under control of the Flight Management system)to provide a LOS pointing capability of a three axis gimbal with thecost, power, and weight savings of a single axis slewable mirror.Coupling with the Flight Management System also will provide greater LOSstability in turbulent air. Pilot and passenger discomfort due tounusual aircraft attitudes is eliminated because the vehicle isunmanned. This innovation will also provide the capability to slew theLOS from side to side while traversing a ground track, thussignificantly increasing the area under surveillance. Allowing the MUAVto fly faster through the use of ground based image management and witha wider footprint obtained by employing a slewable LOS mirrorsignificantly reduces the time to search a 4 square mile area from about5 hours to about 30 minutes.

The camera 11 generates the imagery which is transmitted to the GroundControl Unit for interpretation by the operator. For night, an IR camerais required. However any device which is capable of providing imagerycan be employed at other times for example, Daytime TV, Low Light LevelTV. Synthetic Aperture Radar may also be employed as an alternativeimage detection mechanism.

The Flight Management System 13 manages and controls the processesonboard the air vehicle. It is preprogrammed on the ground using theGCU, but is also capable of being reprogrammed enroute at the operatorsdiscretion. The FMS 13 directs the air vehicle to fly to certainlocations and direct the LOS slewing device to predetermined areas ofinterest. It also computes the vehicle location using an InertialMeasurement Unit (IMU) or Global Positioning System (GPS).

The sensor imagery, air vehicle location, and sensor pointing angle,singularly and collectively referred to as image data are communicatedto the Ground Control Unit via a transceiver 12 located in the airvehicle. Alternatively, determination of Global Position may be carriedout onboard the aircraft and the resulting position data communicated tothe Ground Control Unit (GCU) in a like manner. Geographic position andobject image data are synchronized. For these purposes the events areconsidered to be synchronized if geographic position calculations andobject image frames are generated within 80 milliseconds of each other.

The imagery and data are received on the ground with a transceiver 21similar to that located in the air vehicle. The imagery and data arerouted to the image management processing unit 22 located in the GCU.

FIG. 3 depicts a preferred embodiment of the Image ProcessingArchitecture 22. The image management processing unit's function is topreprocess the imagery, eliminate unwanted clutter objects fromconsideration, and identify potential targets of interest for operatorconsideration. Several algorithms exist for this function which arefamiliar to those skilled in the art. Such algorithms include but arenot limited to: Target detection, Target identification, Targetrecognition, Moving Target detection, Mosaic imaging techniques,Differencing (from last survey) identification. These algorithms areoperator selectable depending on the mission and not all would beemployed simultaneously at a given time. The objective of thesealgorithms is to reduce the number of objects for operatorconsideration, thus reducing his workload and enabling him to accessobject and evaluate image information in real or near real time Theprocessed image is displayed to the operator on the Display and ControlInterface 23. Objects of interest are visually highlighted for hisconsideration. An operator interface is provided to enable selection ofthose objects for which the operator wants to determine their location.The interface may also be configured to immediately alert the operatorof the presence of objects of interest.

FIG. 4 demonstrates a means for calculating Target Location usingMultiple Frame Triangulation and eliminating the need for laser ranging.Multiple Frame Triangulation uses the distance traveled from 24 to 25and LOS angle information obtained from multiple frame sightings totriangulate the object's position. An algorithm is employed which usesthe distance traveled between sightings as the base of the triangleformed between two LOS angles obtained from two sightings to obtain anestimate of the objects geographic position. Multiple frames can beaveraged and filtered to obtain accurate locations independent ofterrain variations.

Downlinked data from the MUAV is received in the communication link 21and routed to the digital memory 22 a. The digital memory serves as therepository for the image and data and serves as a buffer to eliminatetiming problems due to delays encountered in the processing of theimagery.

The Clutter Filter 22 b contains the algorithms previously describedwhich serve to eliminate non targets from targets of interest. Note thatthis filter can be bypassed should the operator wish to view the imagerywithout filtering. Objects of interest are flagged by the Clutter filter22 b for highlighting on the operator's display 23 b and noted in theLOS In Frame Computations module 22 c.

The In Frame Computations module 22 c functions to locate the flaggedobject in the image field of view and compute the objects line of sightdirection with respect to the air vehicle coordinate reference frame.The LOS data are routed to the Multiple Frame Triangulation Computationsmodule 22 f for further processing. The Multiple Frame TriangulationComputation module 22 f uses the technique diagrammed in FIG. 4 toaccurately determine the objects location on the earth without the needfor laser ranging. Multiple Frame Triangulation uses the distancetraveled and LOS angle information obtained from multiple framesightings to triangulate the object's position. The algorithm uses thedistance traveled between sightings as the base of the triangle formedbetween two LOS angles obtained from multiple sightings. Multiple framescan be averaged and filtered to obtain accurate locations independent ofterrain variations.

The operator interfaces with the invention using the man-machineinterface 23 a. The man machine interface permits the operator toidentify objects of interest and control the GCU functions. Theinterface may also be programmed to immediately alert the human operatorto the presence of an object. This feature may be employed when themission is airfield perimeter defense and one or several MUAVs fly theperimeter and notify the operator when objects are detected. Theinterface can also allow the human operator to select an object ofinterest and redirect the sensor to the object's location by returningthe MAUV to the area and manipulating the sensor's LOS. Locking onto theobject is another variant.

The operator may not be able to process all pre selected objects ofinterest in real time in a target rich environment. Accordingly, aplayback speed controller 22 e is provided to permit the operator toslow or fast forward the image display to permit optimal viewing

Other embodiments are within the following claims.

1-14. (canceled)
 15. A remote image data processing system for use in an aircraft comprising: a. an aircraft including an adjustable propulsion and control system and connected to servos; b. an image detector located in the aircraft and oriented to detect images on the ground during flight; c. an apparatus located in the aircraft capable of controllably adjusting the image detector's line of sight with the ground; d. a geographic positioning device located in the aircraft; e. a first radio transmitter located in the aircraft connected to the image detector, line of sight sensor and geographic positioning device and configured transmit images, line of sight angle and geographic position of the sensor from the aircraft as data; f. a first radio receiver located in the aircraft capable of receiving inputs to control the aircraft's flight and directing the orientation of the line of sight apparatus transmitted to the receiver and configured and connected to servos capable of actuating the aircraft's flight controls and the line of sight adjustment apparatus; g. a second radio receiver remote from the aircraft and configured to receive images, line of sight angle and geographic position data transmitted by the first transmitter; h. a second radio transmitter remote from the aircraft capable transmitting control signals to the aircraft to control the aircraft's flight and direct the orientation of the line of sight apparatus; i. a computer connected to the first receiver configured to store, analyze, identify and display images detected by the image detector to a human operator and connected to the second transmitter, the computer being additionally connected to the second transmitter the computer being configured to generate control signals capable of transmission by the second transmitter and receipt by the first receiver to control the aircraft's flight and direct the orientation of the line of sight
 16. A remote image data processing system for use in an aircraft comprising: a. a MAUV including adjustable control surfaces and propulsion systems connected to servos; b. an uncooled infra red camera located in the MAUV and oriented to detect images on the ground during flight; c. an apparatus located in the aircraft capable of controllably adjusting the image detector's line of sight with the ground; d. a GPS located in the aircraft; e. a first radio transmitter located in the aircraft connected to the image detector, line of sight sensor and geographic positioning device and configured transmit images, line of sight angle and geographic position of the sensor from the aircraft as data; f. a first radio receiver located in the aircraft capable of receiving control signals to control the aircraft's flight and directing the orientation of the line of sight apparatus transmitted to the receiver and configured and connected to servos capable of actuating the aircraft's flight controls and the line of sight adjustment apparatus; g. a second radio receiver remote from the aircraft and configured to receive images, line of sight angle and geographic position data transmitted by the first transmitter; h. a second radio transmitter remote from the aircraft capable transmitting control signals to the aircraft to control the aircraft's flight and direct the orientation of the line of sight apparatus; a. a computer connected to the first receiver and transmitter configured to store, analyze, identify and display images detected by the image detector, as well as the image's geographic position to a human operator, the computer also being configured to display at the operator's option, images in unanalyzed format in real time, on a frame by frame basis, at varying speeds to alert the operator when the computer identifies an object of interest, the computer being additionally connected to the second transmitter the computer being configured to generate control signals capable of transmission by the second transmitter and receipt by the first receiver.
 17. A remote image data processing system for use in an aircraft as claimed in claim 16 above wherein an apparatus capable of measuring the angle of the line of sight relative to the image detector sensor attached to the line of site apparatus;
 18. A remote image data processing system for use in an aircraft as claimed in claim 16 above wherein said apparatus for resolving detected images is a motor drive telephoto lens connected to the infra red camera;
 19. A remote image data processing system for use in an aircraft as claimed in claim 16 above wherein said apparatus for adjusting image detector line of sight is a three axis pan and tilt mechanism.
 20. A remote image data processing system for use in an aircraft as claimed in claim 16 above, wherein the control signals actuate the aircraft's flight controls, direct the orientation of the line of sight apparatus, adjust the line of sight adjustment apparatus, redirect the aircraft back to an object of interest and fix the image sensor orientation on the object, the control signals being transmitted in real time as a part of pre-programmed flight instructions.
 21. A method for remote image detection, location and identification for use on an aircraft comprising the steps of: a. detecting images using an image detector, located in an aircraft; b. determining the aircraft's geographic position; c. measuring the angle line of sight of the image detector relative to the object; d. calculating a geographic position of the object relative to the aircraft using the aircraft's geographic position and the line of sight angle and correlating the object geographic location with a matching object image frame; e. analyzing object images using object identification, recognition algorithms to prescreen objects of interest to the operator; f. sorting and collating frames containing objects identified by use of object identification, recognition algorithms for review by the operator.
 22. The method as recited in claim 20, further comprising the step of: viewing the detected images as they are received in real time with out use of object identification/recognition algorithms.
 23. The method as recited in claim 20, wherein the aircraft position location is obtained from GPS data.
 24. The method as recited in claim 20, wherein the aircraft position location is obtained from an inertial guidance system.
 26. (canceled)
 27. (canceled) 